THE MAPPING CLASS GROUP ACTS REDUCIBLY 
ON SU(n)-CHARACTER VARIETIES 



WILLIAM M. GOLDMAN 

Abstract. When G is a connected compact Lie group, and tt 
is a closed surface group, then Hom(7r, G)/G contains an open 
dense 0ut(7r)-invariant subset which is a smooth symplectic man- 
ifold. This symplectic structure is Out(7r)-invariant and therefore 
defines an invariant measure fx, which has finite volume. The corre- 
sponding unitary representation of 0ut(7r) on L 2 (Hom(7r, G)/G,/i) 
contains no finite-dimensional subrepresentations besides the con- 
stants. This note gives a short proof that when G = SU(n), the 
representation L 2 ( Horn (ir, G)/G, fi) contains many other invariant 
subspaces. 

Let G = SU(n) and 7r be the fundamental group of a closed oriented 
surface S. Let Hom(7r, G) be the space of representations tt — > G. The 
group Aut(7r) x Aut(G) acts on Hom(7r,G). Let Hom(7r, G)/G be the 
quotient of Hom(7r,G) by {1} x Inn(G). Then 0ut(7r) := Aut(7r)/lnn(7r) 
acts on Hom(7r, G)/G. The 0ut(7r)-action preserves a symplectic struc- 
ture on Hom(7r, G)/G which determines a finite invariant smooth mea- 
sure [x (on an invariant dense open subset which is a smooth manifold). 
When G is compact, the total measure is finite ( Jeffrey- Weitsman |H1IZ|, 
Huebschmann jS]). There results a unitary representation of 0ut(7r) on 
the Hilbert space 

TL: = L 2 (Hom(7r,G)/G,/i). 

Let C C Tt denote the subspace corresponding to the constant func- 
tions. 

The following theorem is proved in Goldman |3] for n = 2 and 
Pickrell-Xia jSJ E] in general: 

Theorem. The only finite dimensional Out(7r) -invariant subspace of 
H is C. 

According to 0, the only finite-dimensional invariant subspace of 
Tt consists of constants. Let TL$ denote the orthocomplement of the 
constants in Tt, that is, the set of / G Tt such that J fdii = 0. 

The goal of this note is an elementary proof of the following: 
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Theorem. The representation of Out(7r) on H,q is reducible. 

In general for a compact connected Lie group G, the components of 
Hom(7r,L7) and Hom(7r, G)/G are indexed by ni(G') where G' is the 
commutator subgroup. In that case the ergodicity/mixing results of 
[31 El Ej imply that the only invariant finite dimensional subspaces are 
subspaces of the space of locally constant functions, a vector space of 
dimension \vert7Ti(G')\. The method of proving reducibility requires a 
nontrivial element of the center of G. For simplicity, we only discuss 
the case G = SU(n) in this paper. 

I am grateful to Steve Zelditch, Charlie Frohman, and Jeff Hakim 
for helpful conversations, and to David Fisher for carefully reading the 
manuscript and pointing out several corrections. 

1. The center of SU(ra) 

Let 3 = Tj/n denote the center of G, the group consisting of all 
scalar matrices (I where ( n = 1. Then Hom(7r,3) — H l (M;Z/n) acts 
on Hom(7r, G)/G by pointwise multiplication: If p 6 Horn (n, G) and 
u G Hom(7r, 3), then define the action u ■ p of u on p by: 

(1) 7 h^A p{i)u{i). 

Recall the definition [2j of the symplectic structure on Hom(7r, G)/G. 
Suppose that p e Hom(7r, G) is an irreducible representation. By 
Weil jTT] (compare also Raghunathan ^U|), the Zariski tangent space to 
\-\om(n,G) at p identifies with the space Z x {ix, gAdp) of cocycles where 
0Adp is the 7r-module defined by the composition 

7T — G Aut(fl) 

and q is the Lie algebra of G. Since p is irreducible, G acts locally 
freely and Hom(7r, G)/G has a smooth structure in a neighborhood of 
[p] . The tangent space to the orbit G-p equals the space of coboundaries 
-B 1 (7r, 0Adp)- The tangent space to Hom(7r, G)/G at [p] identifies with 
the cohomology group H 1 ^, Qm p )- 

Let u G Hom(7r, 3)- Since Ad (3) is trivial, the action of u induces 
an identification of 7r-modules 0Adp — ► Qkd{u- P ) and hence of tangent 
spaces 

T [p] Hom(vr,L7)/L7 = iJ 1 (7r,gAdp) 

— ► T [u . p] Hom(7r, G)/G = H 1 {-k, 0Ad(«. P ))- 
The symplectic form u p at [p] is defined by the cup product 
H\n,g Mp ) x H\7i,g Mp ) — > H 2 (n;R) 
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using the pairing of 7r-modules induced by an Ad-invariant nondegen- 
erate symmetric bilinear form on g as a coeefficient pairing. Evidently 
the symplectic form u, and therefore the corresponding measure /i, are 
Hom(7r, 3)-invariant. 

2. Trace functions 

Suppose that 7 G 7T. The function 

Hom(7r,C7) — >C 

p 1 — > tr(p( 7 )) 

is lnn(G)-invariant and defines a function 

Hom(7r,G)/G % C. 

We extend this definition to products of more than one element of 
7r: Let 7 = (71, . . . ,7 S ) G 71 s . Define the trace function of 7 as the 
product: 

t 7 ([p]) :=t 7l ([p])...t 7s ([p]) 

Using the definition (JTJ) of the action of Hom(7r, 3) on Hom(7r, G)/G, 
the action of a 6 Hom(7r, 3) on the trace function i 7 is given by u ■ t 7 , 
defined by: 

(2) p^ M (7)" 1 t 7 (p) 
since 

(u-t 7 )(p) := t^u- 1 ■ p) 

= tr((n- 1 -p)( 7 )) 

= tr(«( T )- 1 p(7)) 
= M ( 7 )- 1 tr(p( 7 )) 

= u(7) _1 * 7 (p) 

Since 3 — Z/n, the evaluation 14(7) depends only on the total ho- 
mology class 

[7] := [7i] + --- + [7-]e#i(£;ZA0, 

defined as the sum of the Z/n-homology classes of the 7^. The evalua- 
tion of u G Hom(7r, 3) on 7 is just the natural pairing 

Hom(7r, 3) x Z/n) — ► Z/n = Z 

(«, [7]) 1 — ► u ■ [7] 

where [7] is the total homology class of 7. 
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Lemma 1. If [7] ^ G H 1 (E;Z/n) ! then 

t-^dfi = 0. 



Proof. Since the homology class of 7 in -£/i(X, Z/n) is nonzero, ^(7) 7^ 1 
for some u G Hom(7r,3). Since (w^ 1 )^ = /1, 

t^dfj, = J {t^ o u~ l )d(u~ 1 } ^[i = u^)^ 1 J t^d^i 

by (J2J). Since m(7) _1 7^ 1, the integral is zero as claimed. □ 

An element 7 G 7r is nonseparating simple if it contains a nonsepa- 
rating simple loop which is homotopically nontrivial and not homo- 
topic to <9£. Since such an element has nontrivial homology class 
[7] G .£/i(£,Z/n), Lemma ^ implies: 

Corollary 2. Suppose 7 is nonseparating simple. Then t 1 G H.q. 

For n = 2, Frohman and Kania-Bartoszynska (Tj have calculated 
J trydfj, for separating simple loops 7 in terms of Bernoulli numbers and 
6j-symbols. 

3. Invariant subspaces 

Let W C Ho denote the closure of the span of all t a , where a is 
nonseparating simple loops. (Alternatively, let a run over all elements 
of 7r with nonzero homology class modulo n.) Clearly W is Out(7r)- 
invariant. 

Proposition 3. W is a proper subspace ofH^. 

Proof. Choose 7 G tt s such that: 

• try is not constant; 

• 7 has trivial homology class in Z/n). 

Such elements are easy to find: for example if 7 consists of one non- 
trivial separating simple loop, or if 7 consists of one n-th power, or if 
7 = (71, . . . , 71) where s is a positive multiple of n and 71 is nontrivial. 
Normalize i 7 so that it lies in Tio'. 

We claim that £ 7 is a nonzero vector in 7i orthogonal to W. It is 
nontrivial since t 7 is nonconstant. Furthermore if a G 7r has nonzero 
homology class [a] G ifi(E,Z/n) (for example if it nonseparating sim- 
ple), then t a is orthogonal to t T 
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Since p(a) is unitary, t a = t a -i. Thus 

r r , 



t^ta-idn (by Corollary EJ) 

where r\ = (ji, . . . ,j s , a -1 ). Since 

M = [7] - [a] ^ 0, 
Lemma ^ implies that (t y ,t a ) = 0, as desired. □ 

When 7r is a free group and n — 2, then ergodicity and weak-mixing of 
the action of Out(7r) on Horn (71*, G)/G is proved in Goldman 0j. Exactly 
the same argument as above (with W replaced by the closure of the 
span of t 7 where 7 is an element of a free generating set of tt) shows that 
the unitary representation of Out(7r) on L 2 (Hom(7r, G)/G, fi) contains 
invariant (necessarily infinite-dimensional) subspaces other than the 
constants. 



References 

1. Frohman, C. and Kania-Bartoszynska, J., Shadow world evaluation of the Yang- 
Mills measure, Algebraic and Geometric Topology, 4 (2001), 311-332. 

2. Goldman, W., The symplectic nature of fundamental groups of surfaces, Adv. 
Math. 54 (1984), 200-225. 

3. , Ergodic theory on moduli spaces, Ann. Math. 146 (1997), 475-507. 

4. , An ergodic action of the outer automorphism group of a free group, 

math.DG/0506401 (submitted) 

5. Huebschmann, J., Symplectic and Poisson structures of certain moduli spaces, 
Duke Math J. 80 (1995) 737-756. 

6. JeffreyL. and Weitsman, J., Bohr-Sommerfeld orbits and the Verlinde dimen- 
sion formula, Commun. Math. Phys. 150 (1992) 593-630 

7. , Toric structures on the moduli space of fiat connections on a Riemann 

surface: Volumes and the moment map, Adv. Math. 109, 151-168 (1994). 

8. Pickrell, D. and Xia, E., Ergodicity of Mapping Class Group Actions on Rep- 
resentation Varieties, I. Closed Surfaces, Comment. Math. Helv. 77 (2001), 
339-362. 

9. , Ergodicity of Mapping Class Group Actions on Representation Va- 
rieties, II. Surfaces with Boundary, Transformation Groups 8 (2003), no. 4, 
397-402. 

10. Raghunathan, M., "Discrete Subgroups of Lie Groups," Ergebnifie der Math. 
58, Springer- Verlag Berlin-Heidelberg-New York (1972). 

11. Weil, A., Remarks on the cohomology of groups, Ann. Math. 80 (1964), 149— 
157. 



6 



WILLIAM M. GOLDMAN 



Mathematics Department, University of Maryland, College Park, 
MD 20742 USA 

E-mail address: wmg@math.umd.edu 



